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(57) ABSTRACT 

A content-based video story browsing system is disclosed 
including setting and representing relations between objects 
and places of the story. A content-based video story brows- 
ing system includes the steps of selecting objects and places 
as factors organizing contents of a video story and setting a 
relational graph relating the objects to corresponding places; 
setting a video story section and a representative scene as an 
events of contents development corresponding to the rela- 
tions of the objects and the places; and displaying the video 
story section and the representative scene and its contents as 
browsing information corresponding to the relation selected 
from the relational graph. 
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CONTENTS-BASED VIDEO STORY 
BROWSING SYSTEM 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to a video story browsing 
system and more particularly to a contents-based video story 
browsing system defining and representing the relationship 
of significant factors in a video story. The factors includes an 
object, a place, and/or an event in time. The present inven- 
tion also relates to a video story browser representing an 
overall outline consisting of paragraphs summarizing sig- 
nificant events for a convenient video story browsing. 

2. Discussion of Related Art 

Videojmiwsers and v ideo story summ arization in the 
related jirt generally utilize key frames^to r&pTcsent a large 
a amount of story contents. Thus, it is difficult and/or 
complicated lo Summarize and browse the overall content of 
a movie or drama b^ extracted key fram es or a character- 
baseaUstoxy— browsing. Moreover, presenting contents of 
some scenes is ambiguous, making it difficult browse the 
exact information. 

Although picture images of a drama or a movie progress 
by interconnection of events, key frame browsing, or 
character-based browsing is simply performed without pre- 
sentmgTnyinterc"Uiiuecied iclatiuns of events. A video story 
browsingtecnnique has Deen proposed for summarizing and 
brow sing the ^Y gra 1 ? points by -extractin g keyTraa es jrom 
dynamic jrnages^and graphically showing the time relation- 
sh ip of each key fra mes. This "technique may Deusable for 
a simple video file, but is not suitable for video files of huge 
volume such as a drama or movie with numerous intercon- 
nected events over time. Moreover, because picture images 
of greater volume produce a large num ber of key frames, it 
is difficult to gr aphicall y repre sent key frames. Accordingly 
it is hard for a user to understand the graphically represented 
information. 

Another technique of key frame-based video story sum- 
marization and browsing has been proposed in whi ch classes 
are formed according to a leng t h of a section from which a 
key frame is extra ctecL .Ueneraily, a small number, of key 
frames are extracted from sections oflongerlengths for 
hi gher classes^ and a L grea ter n umner of key frames ar e 
ext racted frornsections of sma ller sections for lo wer class es, 
thereby forming a tree struct ure. However, this structure 
cantioi be" used comprehensively to browse the overall 
contents with representative scenes because the key frames 
are extracted from a too large section for the higher classes, 
making it difficult to retrieve a desired part using the key 
frames. 

Still another object-based browsing technique has been 
proposed in which objects of dynamic images are organized 
and an attribute of each ob ject in a section is d isplayed, 
However, this technique also makes it difficult to understand 
the overall contents. In addition, it is often difficult to 
determine which attribute Thmild bf* rtffiiflmt^ti for pirh 
obje ct during a search algori thm. 

Therefore, since video stories such as a drama or movie 
progress by changing characters and places in time, it would 
not only be impossible to summarize the contents, but would 
be difficult to browse the overall contents just by watching 
extracted characters or scenes. 

SUMMARY OF THE INVENTION 

Accordingly, an object of the present invention is to solve 
at least the problems and disadvantages of the related art. 
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An object of the present invention is to provide a content- 
based video story browsing system. 

Another object of the present invention is to provide a 
video story browsing system summa ria ngjhe co n tenis-o f a 
5 video st ory acc ording to the relations of an object, a pla ce, 
an evenCand time. 

Still anomef^bject of the present invention is to provide 
a content-based video story browsing system allowing a 
browsing by a relational graph reflecting a development of 
10 events in time with respect to a main object and place in a 
movie or drama. 

A further object of the present invention is to provide a 
content-based video story browsing system allowing a 
15 browsing by sections of a movie or drama, each section 
describing events with respect to an object and a place in the 
section of the movie or drama. 

Still a further object of the present invention is to provide 
a content-based video story browsing system allowing a 
20 browsing by relation of a main object and a place in which 
one relation is represented by a frame and description 
corresponding to the section of the story containing the 
relation. 

Still a further object of the present invention is to provide 

25 a contents-based video story browsing system allowing 
browsing a video story based upon contents of a video story 
expressed by relations of a main object, a place, and an 
event, in which relations of plural objects and a place as well 
as the relations of one object and one place can be browsed. 

30 Still a further object of the present invention is to provide 
a contents-based video story browsing system allowing 
summarization and browsing of a video story by displaying 
the contents of the video story in event paragraphs, in the 
form of a text or a text and representative scenes. 

35 Additional, advantages, objects, and features of the inven- 
tion will be set forth in part in the description which follows 
and in part will become apparent to those having ordinary 
skill in the art upon examination of the following or may he 
learned from practice of the invention. The objects and 

40 advantages of the invention may be realized and attained as 
particularly pointed out in the appended claims. 

To achieve the objects and in accordance with the pur- 
poses of the invention, as embodied and broadly described 

45 herein, the present invention discloses a contents-based 
video story browsing system based upon relations of objects 
and places. The contents-ba sed video story browsing systenv 
inc ludes a scheme indicating a video M nr y ^rtinn anH a ] 
rep resentative scene as information for the f^^rm^nt of ■ 

5Q the story even ts..^ 

According to another embodiment of the present 
invention, a contents-based video story browsing system 
includes steps of sele cting objects and places as' factor s 
or ganizing 'the contents of a video story and setting a 

55 relational graph relating the objects to corresponding place s- 
setting a video story section and a representative scene as an 
event o f contents d evelopmen t corresponding to the rela- 
tions rJ~thr. nhje^anH the-places: and displayingjhe video 
sto ry section a nd the representative scene and its contents as 

60 the browsing information corresponding to the relati on 
selected from the relational graph. 

AccoTding to another embodiment of the present 
invention, a contents-based video story browsing system 
includes the steps of setting a relational jrrgp h r>f Tiain 

65 obj ects and places as fa ctors organizing contents of a vi deo 
stpr ^ and adding an event description of text along with' 
position information of sections corresponding to the rela- ~ 
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tionj ind a representative fr ame posi tion information repre- presenting the multiple level of characters and places related 

sen ting each section; displaying an object -place relational to the events; and 

graph of the video story; and if at least one relation is piGS. 14 to 16D illustrate example data structures for a 

selected from the relational graph, displaying a representa- content based browsing system according to the present 

tive fra me o f each s ection i n th e selected relation ancTaH" 5 invention, 
event descr iption b ased upon the position informatio nj and 

scre ening ajiser-desi red section. " DETAILED DESCRIPTION OF PREFERRED 

The contents-based video story browsing system further EMBODIMENT 

includes a step of setting the object -place relational graphs n - . j • , . . c j 

... * * *u ft 'j ci j u j cT a m Reference will now be made in detail to the preferred 

with respect to the overall video files and each video file, and 10 ... . - . . , ~ . 

schematically browsing the overall contents of several video ^odiments of he present mventmn, examples of which 

riles from the relational graphs with respect to the overall *" dlus,rated m ^ «»»P"y«S ***™8». 

video files and browsing in detail each video file. FIG ' 1 1S a flow cnart for developing the database used in 

According two another embodiment of the present a ?i de0 * 0Ws f er - , G ( enera J^ SUCh a k da,a fe by 

invention, the overall outline of a video story is divided by 15 ***** of/^ons between objects and a places 

• , , j , l *• u 111 a video story such as a movie or a drama. As shown in 

significant events and event paragraphs representing each * . , , , A , 

& t j , • j . . ■ -j f FIG. 1, characters and places necessary to develop events in 

events, and a scheme indicating a video story section cor- t 4 . *\ . A . ; 7 , F A . , 

j • . , . u • * tit * • * a story are set as main objects and main places, respectively 

respondmg to each event paragraph is set. If a certain event r t \m\ 

is selected, the information about the corresponding story v P )- 

section is displayed in response to the selection. 20 Subordinate classes to main objects and main pjacesmav 

also be defined . For example, if the main pi are rgrr 

BRIEF DESCRIPTION OF THE ATTACHED "h ospital," scenes may be subdivided as subordinate classes 

DRAWINGS into a corngor, a warcL_a_surg ery room, a lobhy^-and a 

hospital_gar-den. However, if representing the main place 

The invention will be described in detail with reference to ^ simply as a "hospita l" woul d better represent the overal l 

the following drawings in which like reference numerals ou{hno of ^ vide o' s tol£ the' subordinate classes arc com- 

refer to like elements wherein: bincd to form si ^, nne m _ a j n placg jainp.ly thr. "h n^^f^ 

FIG. 1 is a flow chart showing the steps of forming the Mt& i setting t he main objects a nd ma in places, re lation- 

browsmg information for browsing a story according to the ships of the main o5jects t0 the main ^TaTelet (step 

present invention; 30 W2 ) . for example, if character A, character B, and character 

FIG. 2 is an example of a graphic data structure setting the c are set as the main objects; and if a hospital, an indoor, and 

relation of an object and a place according to the present an outdoor are set as the main place, the relations of the 

invention; objects and places may be set as follows: places where 

FIG. 3 is a block diagram for a video story production and character. A ap pears are t he hospitals and the indoo r, places 

a video story browser according to present invention; 35 where character B appears are the ind oor and th e outdoor; 

FIG. 4 is an example of a user interface with a graphic and a P| ace where character C appears is the out door, 

data structure setting the relation of an object and a place The set relationships reflect the sections of the video story 

according to the present invention; in which objects actually appear in the video story. 

, FIG. 5 is another example of a user interface with a 40 Accordingly, the sections in which objects appear with 

graphic data structure setting the relation of an object and a respect to places are defined and indexed using the set 

place according to the present invention; relationships. A section may signify a frame in which an 

FIG. 6 is an example of a user interface when several obje ? . actuaUy u ap f aRi or may signify an interval , of f ™ mcs 

objects and places are selected in a graphic data structure m wh A lcb an ° b J ect generally appears. For example, charac- 

setting the relation of an object and a place according to the 45 * ra A a A ™% alternately appear in a section where 

present invention* characters A and B are carrying on a conversation. Thus, 

' neither characters A or B actually appears for the full 

FIG. 7 is an example of a user interface representing an duration of thc frame mtcrval However, the frame interval 

events in the overall story as a paragraph and representing wou]d be mdexed a a k which eithcr character Aor 

key frames and their time relations according to the present g appears. 

invention, Thereafter, a representative scene for each indexed sec- 

F1G. 8 is an example of a user interface representing a doD are set (step 103) A ftame of each sectioQ defined 

events in the overall story as a paragraph and presenting a according to the relationships set in step 102 which best 

scene representing the events as a key frame according to the repr esents the corresponding section is set as the represen- 

present invention; ^ tative scene jh e f rame location of the representative scene 

FIG. 9 is an example of a user interface representing a is marked. In the example above, a frame showing an 

events in the overall story as a paragraph and presenting the operation may be set as the representative scene of a section 

relationship of characters and places related to the events, in which character A comes into the hospital with an illness, 

and presenting representative scenes according to the and the location of that frame would be marked, 

present invention;, 60 F i na Hy ( the events are illustrated for each section by 

FIG. 10 is an example of a user interface showing addi ng appropri ate notes to corresponding sec t ions f step 

representative scenes by respective event paragraphs form 104). Accordin g^ot ^abov e^xa mple. t he operation frame 

the story according to the present invention; is set as a r epresentative scene *for_th e_reialion of character 

FIG. 11 shows a structure of browsing information for A and the hospital, and notes which summarize the story 

embodying of FIGS. 9 and 10; 65 contents of the section are added as text. 

FIGS. 12 and 13 are each examples of a user interface As described above, the relations of the main objects and 

showing events in the overall story as a paragraph and main places are established based upon the contents of the 
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video story. Moreover, the basic information of object, the digital video data 301 stored in the video buffer 302 as 

place, evejit, -and time for a vide o stor v browsing is complete a multimedia video stream file. A control processing unit 304 

by inHf.TQpg relftv^ni^pwini^fifttiirig representative scenes produces or reads the necessary information from the digital 

in^fiflch section, ..and adding, notes to each section . In the video data 301. A user interface and display unit 305 

preferred embodiment, the developed information is stored 5 displays the video data processed by the video processing 

in the header of a multimedia stream. However, the infor- unit 393 aa d the control processing unit 304, and provides 

mation may be stored in other locations. As a result, the ^ interface to receive an input from a user for production of 

video story browsing system allows users to perform a a browsing scheme. 

content-based browsing of a video story by reading the A , . ' ...... 1 .1 

information stored in the header of the multimedia stream. in . A browsing scheme such as the object-place/eyent rela- 
FIG. 2 shows a graphical data structure of the relations 10 Uon of J FIG ' 2 18 P r0 <*^ by the control process- 
between objects and a place/event developed by the steps m § uml 304 and temporarily stored in a memory for video 
described in reference to FIG. 1. Referring to FIG. 2, information 306. The processed schemes are then stored in 
character A 201, character B 202, and character C 203 a re set a storage 307 in a proper format along with the video data 
as themain nhjgcts; and_a _hospital 204. an indoQr .205. and st ored in the video buffer 302. The video data stored in the 
an outdoo r 206 a re set as the^suu^laces^Character A has 15 storage 307 and the browsing schemes are provided to the 
relati ons "to t he hospital 20 4 and tba -antHnnr 206 character user through a recording media, a wire-broadcasting, or a 
B has relations to the hospital 204 and the indoor 205, and radio -broadcasting network. 

character C has a relation to the indoor 205. The relations of A user browses and views the video data and object-place/ 

the objects 201 to 203 and places 204 to 206 are graphically event data stored in the storage 307. A video buffer 308 

shown by links such as the link 207. 20 temporarily reads and stores the video data from the storage 

The sections in which each character appears are indexed 307. A video processing unit 309 and a control processing 

according to the object-place relations. As shown in FIG. 2, unit 310 processes the video data stored in the video buffer 

three sections 1 to 3 (208, 209 and 210) in which character 308. Particularly, a memory for video information 312 reads 

A appears in the hospital 204 would correspond to the and temporarily stores the object-place/event data from the 

relation between character A and the hospital 204. A scene 25 storage 307. The control processing unit 310 processes the 

representing each of the three sections, the start and the end information from the memory for video information, 

positions of the section, and the event illustration of each Subsequently, the processed video data and information are 

section are also illustrated. displayed according the browsing selections received 

Particularly, in section 2 (209) character A appears from 3Q through a user interface and display unit 311. 

frame 16 to frame 21. Ac cordingly, frame 18 becomes a key Therefore, in the content-based video story browsing 

\ frame 2 11 as^ the rep resentative scene and its position system of FIG. 3, the procedure illustrated in FIG. 1 and the 

1 inf ormation Frm_K is recorde d The f rame 16 is a start resulting data structure of FIG, 2 are completely organized 

frame 111 of section '1 {2097~and its position information through a user interface and display unit 305 by the supplier. 

Frm_S is recorded, and the frame 21 becomes an end frame 35 The database is stored in an appropriate storage 307 along 

213 of section 2 (209) and its position information Frm_E with the video data. A user may then perform video story 

is recorded. A summary of the story contents or event for browsing through a user interface and display unit 311. 

section 2 (209) is stored as text Event(text) 214. FIG. 4 shows an example of a user interface for the user 

A human object scheme 215 which is a lower class of the video story browser according to the present inven- 

subordinate to a main object may be defined for the main ^ tion. The user interface corresponds to the relations of 

objects 201 to 203, and a place object scheme 216 which is object-place/event of FIG. 2. The video story browser 400 

a the lower class subordinate to a main place may be defined displays an object-place relational graph 401 and relation 

for the main places 204 to 206. Namely, the place object contents 402 illustrating the story contents of sections cor- 

scheme 216 may define a ward, a corridor, a lobby, a surgery responding to a selected relation. The object -place relations 

room, etc. as the lower class places of the hospital 204 and 45 developed are as follows: character A 403 has a relation to 

object-place relations would be set according to the defined a hospital 406 and an outdoor 408; character B 404 has a 

lower class places rather than the hospital 204. relation to the hospital 406 and an indoor 407; and character 

The graphic structure shown in FIG. 2 is only a portion of C 405 has a relation to the indoor 4071. 

the object-place/event relations. In an actual structure, there While browsing, it a user selects the relation between 

may be any number of sections with respect to the relation 50 character B and the hospital 406 by selecting a link 409 or 

of one object and one place; there may be any number of by directly selecting the character B and the hospital 406 

objects having relations to one place; and there may be any through the use r interface jm d display unit 311, the control 

number of places having relations to one object. To represent processing unit 310 processes to display the four sections 1 

a scene without any objects, the structure may represent a to 4 (410) and events 411 corresponding to the above 

place having a relation with a null object. There also may be 55 relation, by referring to the memory 312. The four sections 

a multiple object-place relation as well as the single object- 1 to 4 (410) are the representative scenes selected from 

place relation. corresponding sections. Also, the notation uooo °" over the 

FIG. 3 shows a first embodiment of a video story brows- link 409 indicates the four existing sections in the relation 

ing system implementing the database discussed in reference between character B and the hospital 406. However, the 

to FIGS. 1 and 2. To facilitate the explanation of the system, eo number of sections need not be indicated, but if indicated, 

the system for the video supplier and the system for the user any other method may be used. 

have been separately illustrated. The video supplier devel- The user may select objects or a place rather than a 

ops a browsing schemes such as shown in FIG. 2 by the steps relational link 409. If at least one object is selected from the 

in FIG. 1, and the user may browse a video story based upon relational graph, sections with the object(s) can be browsed 

the browsing schemes. 65 and displayed regardless of the places. Likewise, if a certain 

Particularly, digital video data 301 is temporarily stored in place is selected, sections with that place can be browsed 

a video buffer 302, and a video processing unit 303 controls regardless of the objects. 
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FIG. 5 is an example of the user interface when one of the 
sections from sections 1 to 4 (410) corresponding to the 
object -place relation (409) in FIG. 4 is se lected fay _sekGttBg — 
the representative scene corresp ondi n g to section 2 (5 10). 
Thus, if a uscTSelects the relationship between character B 5 
504 and the hospital 506, and then selects section 2 (510), a 
more detailed description 511 of section 2 (510) and the 
representative scene are displayed to the user in the relation 
contents 502. 

The video story browser 500 still displays an object-place 10 
relational graph 501 with the data structure of the characters 
503 to 505, the places 506 to 508, and the selected link 509. 
The user may reproduce a portion of the video data corre- 
sponding to section 2 (510) through an icon 512. In 
response, the start frame to the end frame of the correspond- 15 
ing section are displayed. 

FIG. 6 is an example of a user interface when more than 
one object and a place are selected from the relational graph. 
The video story browser 600 displays an object-place rela- 
tional graph 601 and relation contents 602. The relational 20 
graph 601 includes a data structure of characters 603 to 605, 
the places 606 to 608, and links 609 and 610. If characters 
A and B and a place 606 are selected, representative scenes 
611 an(j jjl3 with illust rations 612 and 614 are displayed. 
The represenlativx-^cenSa ^ l l and 613 -<orresp6nd with 25 
sections _in__wJuch—daafa«ter A andLchaTact er B simu lta- 
neo usly appe^ r£_in-tfle--flQSpital 606. 

As discussed above, if plural objects are selected without 
selecting a place, the video story browser 600 displays 
sections in which the selected objects appear regardless of 
the places. Thus, the user may view the places in which each 
selected object appears. Similarly, if one place is selected 
without selecting an object, the video story browser 600 
displays the sections in which the selected place appears 
regardless of objects. 

FIG. 7 shows an example of a user interface representing 
the events of a story by paragraphs and representing the 
relations of the events, including relations in time, by graph 
in key frames. The video story browser 700 has a story ^ 
display 701 with event paragraphs 701a and when one event 
paragraph is selected, the section contents 702 shows a 
transition graph linking scenes 1 to 4 of the section corre- 
sponding to the selected event. 

FIG. 8 is an exam ple of a user i nterface repre senting 4<; 
e vents of the stcay--b£-i>aWfaffe-aiid[ p resenting scene s 
repr esenting the events in the kev fram es. The video story 
browser 800 has a story display 801 with event paragraphs 
801a and when one event paragraph is selected, the section 
contents 802 shows scenes 1 to 7 of the section correspond- 50 
ing to the selected event with brief description for each 
scene. 

FIG. 9 is an example of a user interface representing 
events of the story by paragraphs and presenting the rela- 
tions between the characters and places as well as represen- ss 
tative scenes of the corresponding event of the story. The 
video story browser 900 has a story display 901 representing 
event paragraphs 901a and when on event paragraph is 
selected, the section contents 902 shows representative 
scenes for a character-place relational graph of a section eo 
corresponding to the selected event. 

A character-place relational graph screen 903 and a rep- 
resentative scene screen 904 for a certain relation are dis- 
played in section contents 902. When one event paragraph 
901a corresponding to a certain event is selected in the story 65 
display 901, the relations of characters A, B and C (905 to 
907) appearing in section corresponding to the selected 
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event and places such as a hospital 908, an indoor 909, and 
outdoor 910, are displayed in the character-place relational 
graph screen 903. If one relation 911 between character B 
and the hospital 908 is selected, representative scenes of a 
section in which character B appears in the hospital 908 are 
displayed 20 in the representative scene screen 904. 

FIG. 10 is an example of a user interface showing 
representative scenes for the respective event paragraphs of 
the story. The video story browser 100(0 has a story display 
1001 representing key frames 14 (or representative scenes) 
and paragraphs corresponding to the events and when one 
event paragraph is selected, the section contents 902 show 
representative scenes for a character-place relational graph 
of a section corresponding to the selected event. A character- 
place relational graph screen 1003 and a representative 
scene screen 1004 of a certain relation are displayed in the 
section contents 1002. 

The browsing method and results displayed in the section 
contents 1002 are the same as described in reference to FIG. 
9. However, in FIG. 10, a„ jepresentative key frame 14 
representing an event may Deselected, rathe r than an event 
paragraph to display the c haracter-place relation and the 
repre sentative scenes corresponding to the selected k ev 
frame 14. 

" FIG. 11 is another example of a user interface represent- 
ing a character-place relational graph according to the 
present invention. Reference numeral 1101 denotes the 
overall story consisting of event paragraphs UOla. Refer- 
ence numeral 1102 denotes scenes representing an event 
section by key frames (or position information) 1102a and 
links 1104 to the event paragraphs of the story section. 

Reference numeral 1103 denotes a graphic structure of the 
relations between characters (1103fl, 1103ft) and places 
(1103c, 1103d) in each section corresponding to the event 
paragraphs 1101a. Character schemes 1103£> subordinate to 
character 1103a are defined as the lower class relational 
schemes to the places 1103c. For example, a 'ward*, a 
'surgery room*, and a 'hospital corridor' may be subordinate 
to the 'hospital*. 

FIGS. 12 and 13 are each examples of a user interface 
showing events of a story by paragraphs and presenting the 
multiple level of characters and places related to the events. 
As shown in FIG. 12, the video story browser 1200 has an 
object-place relational display 1201 and a story display 1202 
representing event paragraphs. The object-place relational 
display 1201 includes a screen 1203 graphically represent- 
ing the relations of objects and places, a representative key 
framej^fl4_£ocresponding to a selected relation, and~a 
description 1205 of the event in the section. 

A representative scene (key frame) representing the event, 
as in FIG. 10, may be displayed in the story display 1202. 
In FIG. 12, three characters A, B and C are set as the objects, 
and a hospital, a character A*s house, a park are set as the 
places. Particularly, character A has a relation to the 'hos- 
pital' and the 'park', and character B has a relation to the 
'hospital', and character C has a relation to character A*s 
home. 

When a relation 1206 between character B and the 
'hospital* is selected, a representative scene of a section in 
which character B appears in the 'hospital* is displayed as a 
key frame 1204 with a description 1205 about the section. A 
display-zoom button 1207 indicates that there is a lower 
class level subordinate to the selected place, namely the 
'hospital*. The display-zoom button 1207 may be provided 
in a menu, rather than being displayed in the section contents 
1201. If a user selects the display-zoom button 1207, ao 
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object-place relation of a level subordinate to 'hospital* 
would be displayed, as shown in FIG. 13. 

Referring to FIG. 13, the video story browser 1300 has a 
section contents screen 1301 and a story display 1302. The 
section contents screen 1301 includes an object-place rela- 
tional graph 1303, a key frame 1304 corresponding to the 
selected relation, and a description 1305 of the event in the 
section. The lower class places defined for the 'hospital* are 
the lobby of the hospital and a ward of the hospital. 
Character B has relations 1306a and 1306b to both the lower 
class places. 

Therefore, a user may select the relation 1306a between 
character B arid The lobby or relation 1306b between 
character B and the ward from a video story browser 1300 
of FIG. 13. In response, a key frame 1304 and its description 
1305 corres ponding to the selected relation woulg^e-dis- 
pla yed. If the button 1307 grouping the lobby and the ward" 
is selected, the screen returns to the upper level relational 
graph as shown in FIG. 12. ^ 

As described above in reference to FIGS. 1 to 13, the 
present invention is capable of browsing a video story by the 
relations of object-place. FIGS. 1 to 13 concern a single 
video file, a single drama or a single movie. 

The present invention employs the concept of the content- 
based video story browsing system. The plural video files 
are represented via a single relational graph, and the contents 
of each video file are represented via a single relational 
graph. The contents can be briefly browsed from the rela- 
tional graph, and a detail browsing may be performed from 
the relational graph for the contents of one video file. If a 
movie is divided into several video files, different events 
develop with the same objects (e.g. the same hero or heroin) 
and same places (the main places of the development of the 
video story). The divided video files are considered as a 
single video file, and its contents are illustrated by the 
object-place relational graph as shown in FIGS. 1 to 13. 
Thus, a user may schematically browse the overall contents 
of the several video files of a single movie. 

FIG. 14 shows an example of the data structure for a story 
browser according to the present invention. As shown, a 
video is divided relation information which includes object 
and place relation information and scene section informa- 
tion. Each scene section information includes time period 
and contents of the section in the story. FIG, 15 shows 
another example of the data structure for a story browser 
according to the present invention. The video is divided into 
scene section information rather than the relation informa- 
tion. Each scene section information inc ludes a time pe riod 
inform ation, conte nts infor mation, and place in formatTonas 
wel l a plurality of object informati on. "" — 

FtGS. 16 A to 16D shows the data structures for the time 
period, object, place and contents information in FIGS. 14 
and 15. Particularly, the time period information includes the 
start frame and the end frame of a section, as shown in FIG. 
16 A. The object information includes name and a represen- 
tative frame, as shown in FIG. 16B, The place information 
includes the name and representative frame, and may option- 
ally include lower class places, as shown in FIG. 16C. The 
contents information includes the text story and the repre- 
sentative frame, as shown in FIG. 16D. 

According to the present invention, the overall contents of eo 
a video story are summarized and a certain position in the 
summary may be selected to browse the detailed contents 
corresponding to that position. In addition, since the present 
invention develops the relational graph based upon the basic 
factors, namely the characters and backgrounds, necessary 65 
for the development of an event, a user can easily understand 
the contents. 



Moreover, this invention may be applicable to a video 
story of animals set as the main objects rather than persons. 
Also, the level of places may be varied with the development 
of the story. Thus, the present invention allows to easily 
5 summarize the contents of the story with development of an 
event since a lower class of an object or place subordinate 
to the main object and place may be defined. 

In addition, the present invention sets the relationship of 
objects, places, and time from the according to the devel- 
10 opment of an event, and various modifications are available. 
For example, for a video story of sports, e.g. golf, the main 
object may be set as a golfer, andjhe m ain pl a ce mly set as 



a group oi 1 a teeing ground, a fa irway, etc Sin™ thA event of 
each scene ma y be related to the kinds o f the golfer's swin gs 
15 the present invention may be effectiv el y implemented in 
~~ sports broadcasting. 

Furthermore, the present' invention include a data struc- 
ture in which the contents of several video files are repre- 
sented via a single relational graph, and another data struc- 
20 ture in which the contents of each video file are represented 
via a single relational graph, such that a user may schemati- 
cally browse the overall contents of the several video files 
from the relational graph and browse in detail the contents 
of a single video file from the corresponding relational 
25 graph, thus assuring a package browsing of the several video 
files. 

According to the present invention, the overall story of a 
drama or movie is illustrated by a collection events in the 
drama or movie, thus removing the shortcomings of the 
conventional key frame-based contents summarization and 
browsing method which describes the development of a 
video story only with representative scenes. Because the 
overall story of a drama or movie is represented by event 
paragraphs, the present invention removes the complexity 
and the difficulty of expressing the overall contents of large 
scale solely by key frames or other conventional browsing 
method, thus ensuring easy and simple summarization and 
browsing of a video story. 

The foregoing embodiments arc merely exemplary and 
are not to be construed as limiting the present invention. The 
present teachings can be readily applied to other types of 
apparatuses The description of the present invention is 
intended to be illustrative, and not to limit the scope of the 
claims. Many alternatives, modifications, and variations will 
be apparent to those skilled in the art. 
What is claimed is: 

1. A content-based video story browsing system compris- 
ing: 

a memory storing a video story and browsing data based 
upon relations between at least two objects and at least 
two places of the video story; and 
a user interface displaying the browsing data, receiving 
user selections based upon the displayed browsing data, 
and displaying a section of the video story correspond- 
ing to the user selections, wherein the displayed brows- 
ing data comprises: 

a relational graph screen graphically showing the rela- 
tions between at least two objects and at least two 
places of the video story, wherein any of said objects 
can be selected with any of said places. 

2. The system of claim 1, wherein the displayed browsing 
data further comprises: 

a relation contents screen showing at least one represen- 
tative scene and brief description of a section in the 
video story corresponding to a selected relation from 
the relational graph screen. 
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3. A system of claim 2, wherein the relation contents 
screen of the user interface shows only one representative 
scene, a display button, and detailed description of a section 
in the video story corresponding to said one representative 
scene when one representative scene is selected from said ay 
least one representative scene; and wherein the relation 
contents screen displays the section in the video story 
corresponding to said one representative scene when the 
display button is selected. 

4. A display of content-based video story browsing system 
of claim 2, wherein links between the at least one object and 
the at least one place are user selectable to correspondingly 
update the at least one representative scene and the brief 
description. 

5. The system of claim 1, wherein the displayed browsing 
data further comprises: 

a relation contents screen showing at least one represen- 
tative scene and brief description of a section in the 
video story corresponding to selected at least two 
relations from the relational graph screen. 

6. A system of claim 1, wherein the displayed browsing 
data comprises: 

a story screen showing at least one event of the video 

story by paragraphs; and 
a section contents screen showing a transition graph 

linking representative scenes of a section in the video 

story corresponding to a selected event from the story 

screen. 

7. A system of claim 1, wherein the displayed browsing 
data comprises: 

a story screen showing at least one event of the video 
story by paragraphs; and 

a section contents screen showing representative scenes 
and brief description of a section in the video story 
corresponding to a selected event from the story screen. 

8. The system of claim 1, wherein the displayed browsing 
data further comprises: 

a story screen showing at least one event of the video 
story by paragraphs; and 

a section contents screen showing a relational graph of the 
relations between at least two objects and at least two 
places of a section in the video story corresponding to 
a selected event from the story screen, and showing at 
least one representative scene of a section correspond- 
ing to a selected relation from the relational graph. 

9. The system of claim 1, wherein the displayed browsing 
data further comprises: 

a story screen showing at least one event of the video 
story by paragraphs and a representative scene corre- 
sponding to each one of said at least one event; and 

a section contents screen showing a relational graph of the 
relations between at least two objects and at least two 
places of a section in the video story corresponding to 
a selected event from the story screen, and showing at 
least one representative scene of a section correspond- 
ing to a selected relation from the relational graph. 

10. The system of claim 1, wherein the displayed brows- 
ing data further comprises: 

a story screen showing at least one event of the video 
story by paragraphs; and 

a section contents screen showing a relational graph of the 
relations between at least two objects and at least two 
places of a section in the video story corresponding to 
a selected event from the story screen, said section 
contents screen also showing at least one representative 



scene and brief description of a section corresponding 
to a selected relation from the relational graph. 

11. A system of claim 10, further comprising a display- 
zoom button indicating a lower class level subordinate to an 

5 object or place in the selected relation, if said lower class 
level exists. 

12. The system of claim 11, wherein the section contents 
screen shows a relational graph of the relations between at 
least two objects and lower class level places subordinate to 

10 said at least two places of a section in the video story 
corresponding to a selected event from the story screen, if 
said display-zoom button is selected. 

13. The system of claim U, wherein the section contents 
screen shows a relational graph of the relations between 

15 lower class level objects subordinate to said at least two 
objects and at least two places of a section in the video story 
corresponding to a selected event from the story screen, if 
said display-zoom button is selected. 

14. The system of claim 1, further comprising: 
a video buffer configured for the temporary storage of 

digital video data; 
a video processing unit configured for controlling the 

digital video data; and 
a control processing unit configured for reading and 
processing the digital video data. 

15. The system of claim 14, further comprising a video 
supplier terminal configured to develop browsing schemes. 

16. The system of claim 15, wherein the video supplier 
terminal comprises: 

a user interface configured to display video data and to 
receive user inputs setting main objects and main 
places, setting relations between said main objects and 
said main places, setting a representative scene in each 
section, and adding descriptive notes to said each 
section; 

a video buffer configured for the temporary storage of 

digital video data; 
a video processing unit configured for controlling the 

digital video data; 
a control processing unit configured for editing the digital 

video data; and 
a memory storing a video story and browsing data based 
upon a relation between at least one object and at least 
one place of the video story. 

17. A method for developing the data structure for a 
content-based video story browsing system comprising: 

selecting at least one object and at least one place as 
factors organizing the contents of a video story and 
setting the selected objects and places as main objects 
and places, respectively; 
setting relations between said main objects and said main 
places, wherein a top level of the data structure initially 
subdivides and organizes said video story by said 
relations; 

setting at least one representative scene of a section in the 

video story corresponding to each set relations; 
setting a composite description for all objects and places 

in each said section in the video story; and 
storing the set information in a storage to be used in a 
content-based video story browsing. 

18. A method of claim 17, further comprising: 
setting low class levels of objects subordinate to at least 

one of the set main objects; and 
setting relations between said low class levels of objects 
and the set main places. 
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19. A method of claim 17, further comprising: 

setting low class levels of places subordinate to at least 

one of the set main places; and 
setting relations between said low class levels of places 

and the set main objects. 

20. A method of claim 17, wherein said representative 
scene indicates a frame position in the video story. 

21. A content-based video story browsing method com- 
prising: 

displaying a browsing data based upon relations between 
at least one object and at least one place of the video 
story; and 

receiving user selections based upon the displayed brows- 
ing data, and displaying a section of the video story 
corresponding to the user selections, wherein display- 
ing the browsing data comprises displaying a story 
screen showing at least one event of the video story by 
paragraphs, and wherein receiving user selections com- 
prises displaying a transition graph linking representa- 
tive scenes of a section in the video story corresponding 
to a selected event from the story screen. 

22. A display of content-based video story browsing 
system comprising: 

a story screen showing at least one event of the video 
story by paragraph; and 

a section contents screen showing a first graphical repre- 
sentation of a section of the video story corresponding 
to a selected event from the story screen, wherein the 
first graphical representation comprises at least two 
objects and at least two places, wherein any of said 
objects can be selected with any of said places and 
wherein the first graphical representative is user select- 
able to display a second graphical representation of the 
user-selected portion of the first graphical representa- 
tion. 

23. A display of claim 22, wherein the section contents 
screen shows a transition graph linking representative scenes 
of the section in the video story corresponding to a selected 
event from the story screen. 

24. A display of claim 22, wherein the section contents 
screen shows representative scenes and brief description of 
the section in the video story corresponding to a selected 
event from the story screen. 

25. The display of claim 22, wherein section contents 
screen shows a relational graph of the relations between at 
least two objects and at least two places of a section in the 
video story corresponding to a selected event from the story 
screen, and show at least one representative scene of a 
section corresponding to a selected relation from the rela- 
tional graph. 

26. The display of claim 22, wherein the story screen 
shows a representative scene corresponding to each one of 
said at least one event, and wherein the section contents 
screen shows a relational graph of the relations between at 
least two objects and at least two places of a section in the 
video story corresponding to a selected event from the story 
screen, and shows at least one representative scene of a 
section corresponding to a selected relation from the rela- 
tional graph. 

27. The display of claim 22, further comprising: 

a section contents screen showing a relational graph of the 
relations between at least two objects and at least two 
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places of a section in the video story corresponding to 
a selected event from the story screen, said section 
contents screen also showing at least one representative 
scene and brief description of a section corresponding 
5 to a selected relation from the relational graph. 

28. A display of claim 27, further comprising a display- 
zoom button indicating a lower class level subordinate to an 
object or place in the selected relation, if said lower class 
level exists. 

10 29. The display of claim 28, wherein the section contents 
screen shows a relational graph of the relations between at 
least two objects and lower class level places subordinate to 
said at least two places of a section in the video story 
corresponding to a selected event from the story screen if 

15 said zoom button is selected. 

30. The display of claim 28, wherein the section contents 
screen shows a relational graph of the relations between 
lower class level objects subordinate to said at least two 
objects and at least two places of a section in the video story 

20 corresponding to a selected event from the story screen if 
said zoom button is selected. 

31. A display of content-based video story browsing 
system of claim 22, wherein said video story comprising a 
textual description for a sequence of events with lineal time 

25 relationship. 

32. A display of content-based video story browsing 
system of claim 22, wherein frames of the single video story 
being viewed have user selectable content. 

33. A system for developing the data structure for a 
30 content-based video story browsing system comprising: 

means for selecting at least one object and at least one 
place as factors organizing the contents of a video story 
and setting the selected objects and places as main 
objects and places, respectively; 

35 means for setting relations between said main objects and 
said main places, wherein a top level of the data 
structure initially subdivides and organizes said video 
story by said relations; 

^ means for setting at least one representative scene of a 
section in the video story corresponding to each set 
relations; 

means for setting a composite description for all objects 
and places in each said section in the video story; and 
45 means for storing the set information in a storage to be 
used in a content-based video story browsing. 

34. A video story browsing system comprising: 

a memory storing a video story and browsing data based 
upon relations between at least one object and at least 
50 one place of the video story; and 

a user interface displaying the browsing data, receiving 
user selections based upon the displayed browsing data, 
and displaying a section of the video story correspond- 
ing to the user selections, wherein the displayed brows- 
55 ing data comprises: 

a story screen showing at least one event of the video 

story by paragraphs; and 
a section contents screen showing a transition graph 
linking representative scenes of a section in the video 
60 story corresponding to a selected event from the 

story screen. 
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